Statistical Regularization and Learning Theory Lecture : 12 Complexity Regularization in Regression

Example 1 To illustrate the distinction between classification and regression, consider a simple, scalar signal plus noise problem. Consider Yi = θ +Wi, i = 1, . . . , n, where θ is a fixed unknown scalar parameter and the Wi are independent, zero-mean, unit variance random variables. Let Ȳ = 1/n ∑n i=1 Yi. Then, according to the Central Limit Theorem, Ȳ is distributed approximately N(θ, 1/n). A simple tail-bound on the Gaussian distribution gives us